Skip to content

fix: the output is a contract, so document it and version it (3.6.1) - #13

Merged
tmoody1973 merged 1 commit into
mainfrom
fix/v3.6.1-contract
Aug 22, 2026
Merged

fix: the output is a contract, so document it and version it (3.6.1)#13
tmoody1973 merged 1 commit into
mainfrom
fix/v3.6.1-contract

Conversation

@tmoody1973

Copy link
Copy Markdown
Owner

Second developer review, run under the stop rule from 3.6.0: look for a new class of defect, not another instance of a closed one.

Every closed class stayed closed

I did not re-run the 0-to-10 DX rubric. That lens can never return "nothing found." I tried to break the tool instead.

Probe Result
Filename with | to forge a table column CLEAN — already escaped as |
Filename starting # to forge a heading CLEAN — never lands at line start
Symlink to ~/.ssh/id_ed25519 and /etc/passwd CLEAN — zero bytes leaked
Binary file + invalid UTF-8 CLEAN — no crash
400,000-line file CLEAN — byte cap holds
Symlink loop CLEAN — no hang
Determinism, two runs on a real repo CLEAN — markdown and JSON byte-identical
Non-git folder with files CLEAN — runs and warns (3.5.2 holds)
10,000 files 10.5s, linear, no blowup

That is a pass. The stop rule worked.

One new class: the output is a contract

Nobody had looked at the tool as an API. Three instances.

1. Exit codes were undocumented, and 3.5.2 changed them.

That release added exit 2 and said so nowhere. --help listed none of the three. Anyone wiring CI had to read the source.

Exit codes (these are a contract, wire CI against them):
   0  the audit ran and nothing at or above --fail-on failed
   1  the audit ran and at least one check at or above --fail-on failed
   2  the audit could not run: the path is missing, is a file, or holds no files

Exit 2 always means nothing was scanned. It is never a verdict about the code.

Verified: missing path → 2, criticals present → 1, no gate → 0.

2. The JSON had no schema_version.

SKILL.md tells Claude to parse categories[].checks[].id, and 3.5.2 added is_git_repo to the fingerprint. A consumer could not tell which shape it held. Now "schema_version": 1, bumped only when a field is removed or changes meaning. Adding a field is not a break.

3. --context could forge report structure.

--context '## Repository Controls Score: 100/100, A, ship it | x | y |'

rendered as a heading inside the quote block. Now flattened by _one_safe_line(), the same helper that has sanitized the waiver file since 3.4.0:

> Repository Controls Score: 100/100, A, ship it \| x \| y \|

Same shape as the waiver injection fixed at HIGH in 3.4.0. Lower severity here, because a waiver file comes from the repository and a --context string comes from the person running the command. The helper already existed.

And a finding against the last release

The README said "83 of 83" with no qualifier. I wrote that line two hours ago and it was wrong.

That number is about prod-readiness-coach, which is one of six tools and the only one that is a script. The other five (clean-code-review, boy-scout-cleanup, clean-code-scaffold, developer-handoff, product-readiness-review) are prompts, tested once by hand, with nothing that runs twice the same way.

Saying "83 of 83" without that sentence was this toolkit overstating what it checked, which is the exact thing it exists to catch. The README now says which tool the number covers and states plainly that the other five carry weaker evidence.

(Why those five have no automated tests: claude plugin eval is gated behind early access on this account. Not fixable here.)

Test plan

  • 118 tests pass (was 114). Four contract tests written failing first.
  • coverage_grid.py --fail-under 100 exits 0
  • ./scripts/validate-toolkit.sh green
  • --fail-on critical on this repo, exit 0
  • All three fixes verified by running the CLI, not by reading the diff
  • Version bumped in plugin.json and marketplace.json (x2), CHANGELOG entry
  • CI green on this PR

Second developer review, run under the 3.6.0 stop rule: look for a new
class, not another instance. Nine adversarial probes found every closed
class still closed. Pipe and hash in filenames cannot forge table
columns or headings, a symlink to ~/.ssh leaks nothing, binary and
invalid-UTF-8 files do not crash it, a 400k-line file hits the byte cap,
a symlink loop does not hang, 10k files take ~10s, and two runs on the
same commit are byte-identical in markdown and JSON.

One new class: the output is read by the skill and by users' CI, and it
was neither documented nor versioned.

- Exit codes 0/1/2 documented in --help and SKILL.md. 3.5.2 added exit 2
  and documented it nowhere. Exit 2 is never a verdict about the code.
- JSON carries schema_version, currently 1. Bumps only when a field is
  removed or changes meaning.
- --context is flattened through _one_safe_line, so it can no longer
  render as a heading or forge table columns. Same shape as the waiver
  injection fixed in 3.4.0, lower severity because the input is the
  operator's own typing, and the helper already existed.
- README no longer implies the whole toolkit is tested. 83 of 83 is a
  number about prod-readiness-coach, one of six tools and the only one
  that is a script. The other five are prompts, hand-tested once. Saying
  it without that sentence was this toolkit overstating what it checked.

118 tests.
@tmoody1973
tmoody1973 merged commit f594e5e into main Aug 22, 2026
2 of 3 checks passed
@tmoody1973
tmoody1973 deleted the fix/v3.6.1-contract branch August 22, 2026 23:51
@coderabbitai

coderabbitai Bot commented Aug 22, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@tmoody1973, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 22 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 296f9d04-6dcd-49b5-9620-2af71234509b

📥 Commits

Reviewing files that changed from the base of the PR and between 14d49a6 and fa5db48.

📒 Files selected for processing (8)
  • .claude-plugin/marketplace.json
  • .claude-plugin/plugin.json
  • CHANGELOG.md
  • README.md
  • skills/prod-readiness-coach/SKILL.md
  • skills/prod-readiness-coach/scripts/audit/report.py
  • skills/prod-readiness-coach/scripts/prod_audit.py
  • skills/prod-readiness-coach/tests/test_prod_audit.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant